Method and system for automatic real-time frame segmentation of high resolution video streams into constituent features and modifications of features in each frame to simultaneously create multiple different linear views from same video source

ABSTRACT

A method, a programmed computer system, for example, a network-based hardware device, and a machine readable medium containing a software program for modifying a high definition video data stream in real time, with no visible delays, to add content on a frame by frame basis, thus simultaneously compositing multiple different customized linear views, for purposes such as creating and broadcasting targeted advertising in real time. The method employs conventional video processing technology in novel and inventive ways to achieve the desired objective by passing data selected by the program back and forth between a GPU and a CPU of a computer. The method is also usable with data streams having lower than high definition where real time processing is desired and yields better results than conventional methods. In such applications, all processing may be done by the CPU of a sufficiently powerful general-purpose computer.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Stage Application of InternationalApplication No. PCT/IL2019/050141 filed Feb. 6, 2019, which claimspriority from U.S. Patent Application No. 62/683,654 filed Jun. 12,2018. The entirety of all the above-listed applications are incorporatedherein by reference.

FIELD OF THE INVENTION

The present invention, in some embodiments thereof, relates toprocessing of high definition video data, and more particularly, but notexclusively, to apparatus and methods for modifying one or more highdefinition video data streams on a frame-by-frame basis to add contentthereto by extracting color mapping information from high-resolutionframes and performing frame modification at the level of the extractedcolor maps, to simultaneously create multiple different synchronizedlinear views from the same original video stream source(s), whilekeeping the source(s) intact for future use. The invention also relatesto non-volatile machine-readable media containing software forimplementing the other aspects of the invention.

The invention has various applications, but will be described in thespecific contexts of systems and methods for adding additional content,for example, targeted advertising, to high definition video data streamson a frame-by-frame basis in real time, i.e., within the presentationduration of the individual frames.

For purposes of the discussion herein, image resolution, including videoframe resolution, is considered high when it is at least HD resolutionof 1920 pixels×1080 lines (2.1 megapixels, aspect ratio 16:9).

BACKGROUND OF THE INVENTION

Computerized video creation and editing, and live video transmission arewell-established and mature technologies in many respects. Commerciallyavailable applications and related support such as Videolicious®,available from Videolicious, Inc. of New York City and OBS Studio, anopen source system, permit live, real time creation of video streamsfrom multiple sources. In that respect, it is an alternative to postproduction editing followed by delayed transmission. Other applicationsthat permit creating composite video streams, even using smart phones asthe input source also exist.

Graphical processing methods and video editing programs capable ofsegmentation of frames into their constituent features are also known inthe art. Such frame processing methods rely on calculations of relationsbetween pixels, which are done by the central processing unit (CPU) anda Graphics Processing Unit (GPU) of an appropriately programmedcomputer. Color segmentation is one such constituent feature, See, forexample, Zitnick, et al., High-quality Video View Interpolation Using aLayered Representation, an application for viewer viewpoint control oflive video streams developed by the Interactive Visual Media Group,Microsoft Research, of Redmond, Wash.

However, known methods require substantial computational time, whichincreases exponentially according to frame size, making these methodsunsuitable for real-time frame processing of high-resolution videostreams, in particular on low-power electronic devices.

Also known, are video-editing programs capable of pixel modification ofa preselected area in a frame. Such frame processing methods applyselected frame processes sequentially, applying each consecutivemodification to the result of its predecessor. However, this processrequires manual editing and compositing of each frame in the video usingthe program, which is a long and expensive post-production professionalprocess.

Also known, are streaming servers which re-stream video while performingthe same frame modifications to all outgoing streams, and televisionediting tables and other specially designed equipment for broadcastingvideo which perform the same.

Also known, are TV set-top-boxes and internet-protocol-based playerswhich can overlay graphics for the viewer to create a unified videoview.

There are, however, no prior art solutions known to the inventors hereofthat provide for real-time high-resolution frame content modificationbased on frame segmentation or any other strategy to produce multipledifferently customized videos synchronously, from the same input videostream(s), at the broadcasting source. The present invention seeks tomeet this need and represents a significant improvement in computerfunctionality for video stream editing applications.

SUMMARY OF THE INVENTION

According to an aspect of some embodiments of the invention, there isprovided method of modifying high resolution video data in real timeusing a computer including a GPU. The method involves receiving thevideo data stream by the computer, extracting color data for theindividual pixels of the frames of the video data stream to create colorlayers, cascading the color layers into a unified color map of featuresin the frame, assembling pixel modification instructions from storeddata according to predefined instructions, creating one or more modifiedsets of video data streams according to the pixel modificationinstructions, output resolution requirements, and desired file formats,while the original source remains intact. and making the modified videodata available for access by one or more intended recipients.

According to some embodiment, the video data is in the form of a videostream provided by any source and in any native format, including adigital camera, a recorded video file, a streaming video file obtainedover the internet, and a commercial video program obtained directlyoff-the-air.

According to some embodiments, the received video data stream is decodedfrom its native format, if necessary, to the internal processing formatemployed by the computer.

According to some embodiments, the pixel modification instructionsinclude modifications of color of the original data. According to someembodiments, the pixel modification instructions include additionalgraphical data to be composited with the original graphical data.

According to some embodiments, the additional graphical data can bevideo data from any source and in any native format which has beendecoded, from its native format, if necessary, to the internalprocessing format employed by the computer performing the method.According to some embodiments, additional graphical data may includeimages, definitions of 3D models, and definitions of text to be renderedby the GPU of the computer.

According to some embodiments the pixel modification instructionstransmitted to the GPU include instructions for adding (compositing)graphical elements. According to some embodiments, the pixelmodification instructions include storyboard script created prior toprocessing of the video stream and read during processing, and/orcommands sent to the system from a human user during processing.According to some embodiments, the pixel modification instructions aregenerated automatically by a computer algorithm external to the computerperforming the method.

According to some embodiments, part of frame, or a whole frame, as colorlayers, is transferred from the GPU to the CPU of the computer forprocessing by the external computer program, wherein the externalprogram is one of a computer vision algorithms and abusiness-intelligence algorithm. According to some embodiments, thecolor layers which will be transferred to the CPU for subsequentprocessing by the external computer program undergo specific processingby the GPU prior to transfer, to prepare them for external logicapplication.

According to some embodiments of the invention, the method furtherinvolves executing a self-calibrating routine by accessing succeedingvideo frames to reduce sensitivity to detection of irrelevant changes inthe video data.

According to some embodiments, the results of external processing areused to automatically generate pixel modification instructions.According to some embodiments, the pixel modification instructions forthe color layers, including additional graphical elements, aretransmitted to the GPU based on stored data for each intended recipientof each modified video stream.

According to some embodiments, the pixel modification instructions forthe color layers include data provided by external systems for eachintended recipient of each modified video stream. According to someembodiments, the one or more modified video data streams are created bymerging the originally segmented color layers and the added graphicalcomponents into a series of single-layer color frames with uniformresolution, one per discrete set of instructions, rendering eachsingle-layer color frame in one or more resolutions, as required byintended recipients, to create final frames, encoding, per resolution,each final frame to one or more codecs to create video components, andmultiplexing, per video component, video and audio components into oneor more different container files.

According to an aspect of the invention, each frame is modified withoutintroducing visible delays to the modified output streams even for highdefinition video data streams, the multiple differently modified videodata streams are created simultaneously and made available to multipleusers, and all created modified video data streams which are based onthe same original video data stream are synchronized with respect to theoriginal video's timeline.

21. According to some embodiments, the incoming video data stream ischaracterized by lower than high definition, and all the processingsteps are performed by the CPU and active memory of a computer withoututilization of a GPU.

A according to an aspect of some embodiments of the invention, there isprovide computer system having one or more central processing units, oneor more graphics processing units, one or more network or othercommunication interface units, a shared memory including mass storagecapability in which there is installed a program for performing a methodas described above. According to some embodiments, the graphicprocessing functions are performed by a GPU or by a CPU operating in aparallel processing mode.

A according to a further aspect of some embodiments of the invention,there is provided a computer readable medium containing a computerprogram operable to perform the method as described above

BRIEF DESCRIPTION OF THE DRAWINGS

Some embodiments of the invention are described below, by way of exampleonly, with reference to the accompanying drawings. It is, however, to berecognized that the particulars shown are by way of example only and forpurposes of illustrative discussion of embodiments of the invention. Inthis regard, the description taken with the drawings makes apparent tothose skilled in the art how embodiments of the invention may bepracticed.

In the drawings:

FIG. 1 is a block diagram of a computer system providing thefunctionality of the present invention according to some embodimentsthereof;

FIG. 2 is an internal process work flow diagram for the systemillustrated in FIG. 1 according to some embodiments of the invention;

FIG. 3 is a flow diagram of synchronization of views creation accordingto some embodiments of the invention.

DETAILED DESCRIPTION OF AT LEAST ONE EMBODIMENT OF THE INVENTION

Introductory Overview:

The present invention, in some embodiments thereof, relates toprocessing of high definition video data and, more particularly, but notexclusively, to apparatus and methods for modifying one or more highdefinition video data streams on a frame-by-frame basis to add contentthereto by extracting color mapping information from high-resolutionframes and performing frame modification at the level of the extractedcolor maps, to simultaneously create multiple different synchronizedlinear views from the same original video stream source(s), whilekeeping the source(s) intact for future use. The invention also relatesto non-volatile machine-readable media containing software forimplementing the other aspects of the invention.

The invention has a variety of applications but will be described by wayof example in the specific contexts of systems and methods for addingadditional content, for example, targeted advertising to high definitionvideo data streams on a frame-by-frame basis in real time, i.e., withinthe presentation duration of the individual frames.

The method according to the invention can also be used for improvedreal-time modification of video data on a frame by frame basis wherehigh definition as defined above is not needed.

In simplest terms, the present invention uses an application softwareprogram to modify the individual pixels in a video data stream withinthe individual frames based on feature segmentation. According to anaspect of the invention, the feature upon which the segmentation isbased is pixel color. According to some embodiments, the segmentation isperformed by a GPU under control of the programming and the CPU of aconventional microprocessor-controlled computer.

More particularly, the present invention addresses the problem thatprocessing of increasingly higher resolution images requires anexponential increase of computing power, so it is impossible to performthe described graphic modifications in real-time on UHD frames on a CPUalone, such that multiple different synchronized views are createdsimultaneously, based on the same source with different added graphicalelements. While it is possible to perform graphic modifications in realtime on HD frames, even this requires a very powerful CPU not typicallyfound in most computers.

According to an aspect of the invention, the different modifications aredone simultaneously to create multiple different views. According tosome embodiments of the invention, the different modified video viewscan be immediately broadcasted to different viewers. According to someembodiments of the invention, the different modified video views can besaved as files. According to an aspect of the invention, the originalvideo source(s) data remains intact and can be reused.

According to an aspect of the invention, there is provided a programmingmodel for a general-purpose computer which significantly improves itsfunctionality in real-time processing of a video stream andframe-by-frame modification of the video stream according to selectablecriteria.

According to some embodiments, the criteria comprise the identity of theintended recipient of the modified video stream.

According to some embodiments, the modification to the video datacomprises targeted advertising. Such targeted advertising can be basedon collected demographic information about the recipient and/or derivedfrom a computer readable medium or information stored in the hostcomputer.

According to some embodiments, the modification instructions can bederived from what is known in the art as a “storyboard script”, i.e., aprogram written prior to processing and read during processing, andcommands, include the definition of the added graphical content.

According to some embodiments, the content of the advertising isdetermined by a human user, as part of the initial configuration of thesystem or in real-time during processing and broadcasting.

According to other embodiments, the added content may be determined byan external system or organization, which makes such determinations byprocessing available data about the intended recipient. The content ofthe advertising can also be determined by non-human, businessintelligence and artificial-intelligence systems, which analyzeavailable recipient data and make decisions based on their algorithms.

According to some embodiments, the programming operates the computer toautomatically create color-based segmentation of a high-resolution videostream in real time, whereby the video stream can be modified accordingto one or more predetermined criteria to add desired modifications toeach frame at the level of the color-based segments, within timeframelimits of a real-time video stream.

According to some embodiments, the program includes a plurality ofdefinitions of frame modifications, a list of mathematical functions,which implements the aforementioned modifications, rules foroptimization of the mathematical functions and rules for application ofthe mathematical functions to assemble the respective framemodifications. The rules for assembly of the mathematical functions areconfigured to assemble all relevant mathematical functions, in acascading manner, into a single composite process.

As used herein, “cascading manner into a single composite process” meansthat the mathematical functions are arranged so the input of consecutivefunctions is the output of their respective predecessors, so theassembly of mathematical functions becomes a single calculation which iseasily optimized by a common routine performed by a GPU, as opposed tolist of discrete functions to be applied one after another.

According to a further aspect of the invention, there is provided anon-volatile machine-readable medium, containing programminginstructions for operating a computer to modify a video data stream inreal time to insert selected content therein on a frame by frame basis.According to some embodiments, the programming instructions implementthe image processing functions described above.

According to a further aspect of the invention, there is provided amethod of modifying a video data stream in real time to add selectedcontent thereto on a frame by frame basis. According to someembodiments, the method implements the image processing functionsdescribed above.

According to yet a further aspect of the invention, there is provided amethod of synchronizing the creation of different views which are basedon the same source, such that while the views differ in file format,resolution, and/or graphic modifications, they do not differ in timelinewith respect to the original source.

Description of an Illustrative Embodiment

Referring to FIG. 1, there is shown a block diagram of a preferredsystem, generally denoted at 10, for implementing some embodiments ofthe invention. The system includes one or more input sources 12 for oneor more high resolution (HD or higher) video files, amicroprocessor-controlled computer system generally designated at 14,and an output interface, generally indicated at 16. Input sources 12 canbe a digital camera, a recorded video file, a streaming video fileobtained over the internet or a commercial video program obtaineddirectly off-the-air.

Computer system 14 may be of conventional architecture and comprised ofconventional components but is programmed according to the invention toprovide a technical solution resulting in enhanced functionality asdescribed herein not achievable by previously known methods and systems.

One particularly advantageous application of the invention is theprocessing of HD or higher resolution video files, in real time, toembed into a succession of frames thereof, additional information, e.g.,an advertising message, that can be targeted specifically towardindividual recipients or groups of recipients sharing common interestsor other characteristics. Accordingly, output interface 16 providesmultiple synchronized targeted output video streams, by way of example,four being shown at 16 a-16 d.

Structurally, computer system 14 may be of any type suitable forhigh-speed video processing. In some exemplary embodiments, such asystem may include an incoming video input interface 18, such as networkinterface circuitry or video card, which receives the incoming videosignal from source 12, a graphics processing unit (GPU) 20, a centralprocessing unit (CPU) 22, a shared high-speed memory 24 which willtypically include mass storage capability, and an outgoing video outputinterface unit 26 which provides the multiple outgoing video stream 16a, 16 b, etc.

The video stream may be derived from any known or hereafter developedsource, for example, a server, a digital camera, a mobile hand-helddevices (smartphones), phablets, tablets, off the air televisionbroadcasts, streaming video provided over the internet or videoprogramming stored on a user's personal computer.

The modified output video stream may be directed to any of the kinds ofdestinations described above as sources of the original video data, aswill be understood by those skilled in the art.

FIG. 2 illustrates an internal process flow diagram for a system such asshown in FIG. 1 according to some embodiments of the invention. Theprocess begins at 30 with input of a high resolution (HD) video datastream, sometimes referred to as a high definition video file or camerastream, for example, from any of the possible sources mentioned above.At 32, the video file is transcoded from its native format to theinternal processing format employed by the system, for example, “raw”video, or alternatively YUV420, RGBA 32 bit.

At 34 GPU 20 is operated to layer each graphical frame into separatepreliminary layers according to color. The GPU optionally executes aself-feedback routine by accessing succeeding video frame(s) to tune GPUlayer sensitivity. This helps eliminate false detection (light changes,stream quality etc.) Thus, it is preferably employed if the incomingvideo is a raw stream from a camera (as opposed to video which waspost-processed with color calibration and smoothing). In such raw datastreams, light artifacts may be present. Weighted average of severalconsecutive frames enables elimination of such artifacts and subsequentcalculation errors.

The program can invite the option to be turned on\off manually by theuser or deduced automatically from selection of input (on for livecamera and off for video from file). By calculating a weighted averageper pixel of colors in a current frame and colors in previous frames,unwanted artifacts such as specks of dust disappear from view.

At 36, two processes proceed in parallel. At 36 a, GPU 20 is operated toperform pixel color modifications and, at 36 b, if an external computerprogram requires this, the GPU creates a duplicate of the original videostream and performs initial preprocessing on a duplicate and sends theresult to the CPU.

An important feature of some embodiments of the invention is the mannerin which the color layer data is provided (between 36 b and 40 b) fromGPU 20 to CPU 22 such that the data is optimized for analysis.

Referring still to FIG. 2, at 38, GPU 20 renders additional graphicalelements by instructions generated from CPU 22 for one or more of thepreliminary color layers; the instructions are created by the CPU from apredefined program or from data received from an external computerprogram. The data added to the individual frames is provided by a datastorage source 40 a containing a database of pre-defined graphicalcomponents and user related instructions for creating the desiredrecipient-specific views, or by an external computer program whichcombines user-intelligence with a database of graphical components 40 b.

At 42, the programming operates GPU 20 to merge the color layers into acomposite frame (comprised of the original frame and added graphicalcomponents, as described in 38) into a single-layer color frame withuniform resolution. The merging is done with respect to optionaltransparency of each layer. The act of merging several graphical layersinto one, also called “flattening” is known to users of graphicalediting software that enables layering, such as Photoshop® and Gimp. Theresult of the merge done by this invention matches the expectations ofthose users.

Then, at 44, GPU 20 can apply an addition frame color palettemodification to create a unified frame by modifying the color of theframe created in 42. As a result, even though the graphics componentscome from different sources, i.e. the main video stream and the composedelements, such as other video streams and added graphics (e.g. CGI),they undergo the same specific modifications, and the outcome seems tothe observer to have come from a single source.

In computers having sufficiently powerful CPUs, and enough activememory, instead of utilizing the GPU to perform parallel video streamprocessing, the program can be designed such that the CPU operates in acontinuous loop over all the pixels within the frame. This option may beuseful where real-time video stream processing is desired, but theincoming video stream has less than HD resolution.

Finally, at 46, GPU 20 renders multiple outputs of new video componentstreams with respect to intended recipient device resolution. Thepractice of outputting several resolution streams per video is known inthe art. Each of the video component streams is encapsulated into adifferent file container (three of which are indicated at 46 a, 46 b,and 46 c), which are then made available to intended recipients 48 a, 48b, and 48 c.

An important feature of some embodiments of the invention is the mannerin which different views of the same source video stream are madeavailable synchronously. The software created according to the presentinvention that provides the real-time synchronization of video streams(whether original or preprocessed) is described in connection with FIG.3 below.

FIG. 3 is a program flow diagram illustrating implementation of theinvention according to some embodiments, to synchronize video streamcreation, such that any number of video stream which are created fromthe same source by application of different pixel modifications willdisplay the same base frame at the same time, regardless of performedmodifications or processing start time. In considering the followingdescription, it is to be understood that the code defining the programmay vary greatly without departing from the concepts of the invention,and that numerous variations of such code will be apparent to thoseskilled in the art.

Turning now specifically to FIG. 3, at 60, the high-resolution videodata is received from an input device such as described above and isdemultiplexed into video and audio components. It is to be understoodthat multiple video streams can be demultiplexed in parallel.

At 70 and 80, two processes proceed in parallel. At 70, video componentstream is received by GPU 20, such that a part of GPU internal memoryand processing power is dedicated to applying a discrete set ofinstructions to a video component (three of such parts are indicated at20 a, 20 b, and 20 c). It is to be understood that when multiplededicated parts are configured to process the same input source, thengraphical frame data is made available to each and all of them at thesame time.

At 80, audio component stream or streams are passed through audiofilters which combine multiple audio component streams into one. It isto be understood that when only a single audio component stream ispassed, it is passed without combination.

At 90, video stream component is decoded from its codec to raw formatand is passed as new texture (i.e., color data layers derived from eachvideo frame) for application of various pixel modifications, includingaddition of graphical components, at 100. Although the preferredembodiment of this invention utilizes GPU 20 to perform decoding, it isunderstood by those skilled in the art that the decoding can also bedone by a CPU 22.

At 110, the final frame is rendered at one or more resolutions and eachframe, by resolution, is encoded at 120. Although the preferredembodiment of this invention utilizes GPU 20 to perform encoding, it isunderstood by those skilled in the art that the encoding can also bedone by a CPU 22.

Finally, at 130, video streams and audio streams are multiplexed intoone or more file containers (three of which are indicated at 140 a, 140b, and 140 c), such that each file container contains an audio streamand a video stream which was modified according to initial configurationof the system in relation to this output. It is to be understood thatwhen multiple outputs, which differ in file formats, resolution, and/orpixel modifications, are based on same source inputs, then the outputsare synchronized.

According to the present invention, these technical capabilities areused in a unique way to achieve the enhanced functionality ofsynchronously creating multiple modified HD video files in real time,i.e., without noticeable delay.

General Interpretational Comments:

Unless otherwise defined, all technical and/or scientific terms usedherein have the same meaning as commonly understood by one of ordinaryskill in the art to which the invention pertains.

It is to be understood that the invention is not necessarily limited inits application to the details of construction and the arrangement ofthe components and/or methods set forth in the above description and/orillustrated in the drawings. The invention is capable of otherembodiments or of being practiced or carried out in various ways.

The terms “comprises”, “comprising”, “includes”, “including”, “having”and their conjugates mean “including but not limited to”. The term“consisting of” means “including and limited to”.

The term “consisting essentially of” means that the composition, methodor structure may include additional ingredients, steps and/or parts, butonly if the additional ingredients, steps and/or parts do not materiallyalter the basic and novel characteristics of the claimed composition,method or structure.

As used herein, the singular form “a”, “an” and “the” include pluralreferences unless the context clearly dictates otherwise.

The described methods and the required programming can be executed onany computer, which incorporates a suitable central processing unit(CPU) to execute the program, and a GPU in those embodiments for whichit is utilized (for high-resolution video streams) to execute thecomplex frame processing, means for acquiring high-resolution inputvideo-streams, as well as active memory and mass storage, for holding acollection of predefined features.

Implementation of the method and/or system of embodiments of theinvention can involve performing or completing some tasks such asselection of intended recipients, manually, automatically, or acombination thereof. Moreover, according to actual instrumentation andequipment of embodiments of the method and/or system of the invention,several selected tasks could be implemented by hardware, by software orby firmware or by a combination thereof using an operating system.

For example, hardware for performing selected tasks according toembodiments of the invention could be implemented on a general-purposecomputer suitably programmed as described herein, or as one or morespecial-purpose chips or circuits, i.e., one or more ASICs.

Although the invention has been described in conjunction with specificembodiments thereof, it is evident that many alternatives and/ormodifications and variations will be apparent to those skilled in theart. Accordingly, it is intended to embrace all such alternatives,modifications, and variations that fall within the spirit and broadscope of the appended claims.

What is claimed is:
 1. A method of automatically modifying highresolution video data in real time using a computer including a GPU, themethod comprising: receiving a video data stream by the computer,wherein the video data stream includes multiple frames; extracting colordata for the individual pixels of the frames of the video data stream tocreate color layers; cascading the color layers into a unified color mapof features in the frame; assembling instructions to modify pixels ofthe frame at a level of the unified color map of features from storeddata according to predefined instructions, wherein the instructions atleast include additional graphical data to be composited with originalgraphical data of the video data stream; creating multiple modified setsof video data streams according to the instructions, output resolutionrequirements and desired file formats, while original source of thevideo data stream remains intact, wherein the modified sets of videodata are different from each other and the creating includes: mergingthe color layers and the additional graphical data into single-layercolor frames with uniform resolution, wherein each single-layer colorframe is based on a discrete set of the instructions, rendering eachsingle-layer color frame based on the output resolution requirements tocreate at least one final frame, encoding, based on the resolution, eachfinal frame to at least one codec to create video components, andmultiplexing the video components into the desired file formats; andmaking the modified set of video data streams available for access byone or more intended recipients.
 2. The method according to claim 1,wherein the video stream is provided by any source and in any nativeformat, including a digital camera, a recorded video file, a streamingvideo file obtained over a network, and a commercial video programobtained directly off air.
 3. The method according to claim 2 furtherincluding decoding the received video data stream from its native formatto an internal processing format employed by the computer.
 4. The methodaccording to claim 1, wherein the instructions include modifications ofcolor of the original graphical data of the video data stream.
 5. Themethod according to claim 1, wherein the additional graphical dataincludes other video data from any source and in any native format, andhas been decoded, from its native format to an internal processingformat employed by the computer by which the method is being performed.6. The method according to claim 1, wherein the additional graphicaldata includes images, definitions of 3D models to be rendered by theGPU, and definitions of text to be rendered by the GPU.
 7. The methodaccording to claim 1, wherein the instructions transmitted to the GPUinclude instructions for adding graphical elements.
 8. The methodaccording to claim 1, wherein the instructions include storyboard scriptcreated prior to processing of the video data stream and read during theprocessing, and/or commands sent from a user during the processing. 9.The method according to claim 1, wherein the instructions are generatedautomatically by a computer algorithm external to the computerperforming the method.
 10. The method according to claim 1, wherein partof frame or a whole frame, as color layers, is transferred from the GPUto a CPU of the computer for processing by computer vision algorithmsand/or a business-intelligence algorithm.
 11. The method according toclaim 10, wherein the color layers which will be transferred to the CPUfor subsequent processing undergo processing by the GPU prior totransfer, to prepare them for external logic application.
 12. The methodaccording to claim 10, wherein results of the processing are used toautomatically generate the instructions.
 13. The method according claim1, further including executing a self-calibrating routine by accessingsucceeding video frames to reduce sensitivity to detection of irrelevantchanges in the video data.
 14. The method according to claim 1, whereinthe instructions for the color layers, including additional graphicalelements, are transmitted to the GPU based on stored data for eachintended recipient of each modified video stream.
 15. The methodaccording to claim 14, wherein the instructions for the color layersinclude data provided by external systems for each intended recipient ofeach modified video stream.
 16. The method according to claim 1, whereineach frame is modified without introducing visible delays.
 17. Themethod according to claim 1, wherein the multiple modified video datastreams are created simultaneously and made available to multiple users.18. The method according to claim 17, wherein modified video datastreams which are based on the original graphical data of the video datastream are synchronized with respect to the video's timeline.
 19. Themethod according to claim 1, wherein the video data stream ischaracterized by lower than high definition, and the steps of claim 1are performed by a CPU and active memory of the computer withoututilization of the GPU.
 20. A computer system for automaticallymodifying high resolution video data in real time, comprising: one ormore central processing units and/or one or more graphics processingunits; one or more network or other communication interface units; ashared memory including mass storage capability; wherein the memorycontains a program that configures the computer system to: receive avideo data stream, wherein the video data stream includes multipleframes; extract color data for individual pixels of the frames of thevideo data stream to create color layers; cascade the color layers intoa unified color map of features in the frame; assemble instructions tomodify pixels of the frame at a level of the unified color map offeatures from stored data according to predefined instructions, whereinthe instructions include additional graphical data to be composited withoriginal graphical data of the video data stream; create multiplemodified sets of video data streams according to the instructions,output resolution requirements and desired file formats, while originalsource of the video data stream remains intact, wherein the modifiedsets of video data are different from each other and the creationincludes: merging the color layers and the additional graphical data ofthe video data stream into single-layer color frames with uniformresolution, wherein each single-layer color frame is based on a discreteset of the instructions, rendering each single-layer color frame basedon the output resolution requirements to create at least one finalframe, encoding, based on the resolution, each final frame to at leastone codec to create video components, and multiplexing the videocomponents into the desired file formats; and make the modified sets ofvideo data streams available for access by one or more intendedrecipients.